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Abstract 

We demonstrate that a reproducing kernel Hilbert or Banach space of functions 
on a separable absolute Borel space or an analytic subset of a Polish space is sepa¬ 
rable if it possesses a Borel measurable feature map. 

1 Introduction 

Reproducing kernel Hilbert spaces (RKHS), see e.g. Berlinet and Thomas-Agnan [2] 
and Steinwart and Christmann [33, Sec. 4], are important in Statistics and Learning 
Theory. Moreover, when using these spaces in Probability and Statistics, separabil¬ 
ity has powerful effects. For example, for any separable metrizable space X we have: 
B{X xX) = B{X)xB{X) for the Borel cr-algebras [12, Prop. 4.1.7], the Ky-Fan met¬ 
ric can be defined so as to metrize convergence in probability [12, Thm. 9.2.2], conver¬ 
gence in probability implies convergence in law [12, Prop. 9.3.5], convergence in law 
is metrized by the Prokhorov metric [12, Thm. 11.3.3], the space of probability mea¬ 
sures with the weak topology is separable and metrizable [1, Thm. 15.12], and the 
Kantorovich-Rubinstein and Strassen theorems have sharp forms [12, Thms. 11.8.2 
& 11.6.2]. Moreover, separable Hilbert spaces are Polish so that we have all the 
machinery of descriptive set theory available, regular conditional probabilities exist 
[12, Thm. 10.2.2], Bochner integration is simple [1, Lems. 11.37 & 11.39], and all 
probability measures on them are tight [11, Thm. 69, 77-BI]. Most importantly, by 
a classical result, see e.g. Halmos [19, Prob. 17], all separable Hilbert spaces are 
isomorphic with ^^(N). 

According to Montgomery [27], “Separability is a property which greatly facil¬ 
itates work in metric spaces, but it may be of some interest to point out that this 
property has been unnecessarily assumed in the proofs of certain theorems concern¬ 
ing such spaces and concerning functions defined on them.” Indeed, many works 
do assume separability of the RKHS. For example, Steinwart and Christmann’s [33, 
Thm. 7.22] oracle inequality for SVMs, Christmann and Steinwart [8, Thms. 7 & 
12], [7], [6], Steinwart and Christmann [34], [35], De Vito, Rosasco and Toigo [9], 
Hable and Christmann [18, Thm. 3.2], Lukic and Beder [25], Steinwart [32] and 
Vovk [38, Thm. 3]. De Vito, Umanita and Villa [10] assume it in their generaliza¬ 
tion of Mercer’s theorem to matrix valued kernels, and Christmann, Van Messem 
and Steinwart [8] assert that Support Vector Machines (SVMs) are known to be 
consistent and robust for classification and regression if they are based on a Lips- 
chitz continuous loss function and on a bounded kernel with a separable reproducing 
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kernel Hilbert space which is dense in where /r is the marginal distribution 

of the data-generating distribution. Cambanis [3] proves that a stochastic process 
with index set a Borel subset of a Polish space has a measurable modification if 
and only if the reproducing kernel corresponding to the autocorrelation function is 
measurable and its corresponding RKHS is separable, and that a second order pro¬ 
cess with index set the real line is oscillatory if and only if its RKHS is separable. 
Nashed and Walter [28] require a separable RKHS in their development of sampling 
theorems for functions in reproducing kernel Hilbert spaces, and Zhang and Zhang 

[41] in reproducing kernel Banach spaces. Hein and Bousquet [22] require it and 
give some sufficient conditions for it. For an example of a non-separable RKHS, see 
Canu, Mary, and Rakotomamonjy [4, Ex. 8.1.6]. 

Let us now briefly discuss the topological spaces under consideration. A Polish 
space is a separable completely metrizable space and a Suslin space a Hausdorff 
continuous image of a Polish space. Following Frolik [16], a metrizable space X is 
said to be absolute Borel if A C Z is a Borel subset for all metrizable Z for which it is 
a subspace. Moreover, Frolik [15] introduces bianalytic spaces as Suslin spaces such 
that their complement in their Cech compactification is also Suslin and, in Frolik 
[15, Thm. 12], shows that a metrizable space is separable absolute Borel if and only 
if it is bianalytic. On the other hand, a subset of a Polish space is called analytic 
if it is Suslin. Indeed, the two types of spaces considered here, separable absolute 
Borel spaces and analytic subsets of Polish spaces, are very general. For example, 
for a Borel subset of a Polish space, Frolik [16, Thm. 1] asserts that it is separable 
absolute Borel and the famous Suslin theorem, see e.g. Kechris [23, Thm. 13.7] 
asserts that it is analytic. That is, they both include any Borel subset of a Polish 
space, in particular any Borel subset of a separable Banach space, so any Borel, open, 
or closed subset of R". Since R" is Polish, it follows that this class also includes 
any analytic subset of R". Counter examples to these spaces include non-separable 
spaces, non-metrizable spaces and non Suslin spaces. Moreover, Lusin’s Theorem, 
see e.g. Kechris [23, Thm. 21.10], asserts that all analytic subsets of a Polish space 
are universally measurable, that is for every cr-finite measure it is trapped between 
two Borel subsets of the same measure. Consequently, any subset of a Polish space 
which is not universally measurable is a counterexample. 

Reproducing kernel Hilbert spaces are Hilbert spaces of real-valued functions 
such that pointwise evaluation is continuous. In their generalization to reproducing 
kernel Banach spaces (RKBS), Zhang, Xu, and Zhang [40] stipulate that a RKBS on 
a set A is a reflexive Banach space of real valued functions on A whose dual space is 
isometric with a Banach space of functions on A, such that pointwise evaluation is 
continuous for both the Banach space and its dual. They then proceed to develop the 
theory much along the lines of RKHSs. In particular, in [40, Thm. 2] they show that 
RKBSs possess reproducing kernels. Moreover, in [40, Thm. 3] they demonstrate 
that if $ : A —^ W is a map to a reflexive Banach space W and 4)* : A —^ W* is a 
map to its dual such that the linear span of the image of each map is dense, then a 
RKBS is determined with reproducing kernel K{x,x') = [4>*(a:), 4>(a;')], where [, ] is 
the dual pairing between W* and W. Moreover, in [40, Thm. 4] they assert that all 
RKBSs possess such maps. Consequently, we refer to such maps $ : A —^ W and 
<I>* : A —>■ W* as primary and secondary feature maps for the RKBS. 

This generalization to RKBSs has generated much interest lately, for example 
see recent results of Fukumizu, Lanckriet and Sriperumbudur [17], Zhang and Zhang 

[42] , Fasshauer, Hickernell and Ye [13] in Machine Learning, in particular of Song, 
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Zhang and Hickernell [30] on sparse learning, and the recent results of Zhang and 
Zhang [41], Han, Nashed, and Sun [20] and Christensen [5] concerning sampling 
expansions, frames and Riesz bases in Banach spaces. 

It is the purpose of this paper to establish separability for both RKHSs and 
RKBSs when the domain is a separable absolute Borel space or an analytic subset 
of a Polish space, in particular when it is a Borel subset of a Polish space, under the 
simple assumption that the reproducing kernel space possesses a Borel measurable 
feature map. 


2 Main Results 

Before our main results, we review some existing results regarding the separability 
of RKHSs. We will consider both when X is not a topological space and when it 
is. When X is not topological, Berlinet and Thomas-Agnan [2, Thm. 15, pg. 33] 
shows that a RKHS H is separable if there is a countable subset Xq C X such that 
f G H and f{x) = 0,x G Xq implies that / = 0. Moreover, a result of Fortet [14, 
Thm. 1.2] asserts that a RKHS with kernel k is separable if and only if for all e > 0 
there exists a countable partition Bj,j G N of A such that for all j G N and all 
xi,X 2 G Bj we have 


k{xi,xi) + k{x 2 ,X 2 ) — k{xi,X 2 ) — k{x 2 ,xi) < e. 

Regarding the separability of RKBSs, an if and only if characterization is ob¬ 
tained through a generalization of Fortet’s Theorem from RKHSs to RKBSs. We 
suspect the proof of our version. Theorem 2.2, is similar to Fortet’s [14, Thm. 2.1] 
for RKHSs, but it is not written down there. Indeed, Fortet’s result is a regularity 
condition on the pullback (pseudo) metric 

d^{xi,X2) ■■= ||4>(a:i) - $(x2)[[ffi = \/k{xi,Xi) -f k{x2,X2) - k{xi,X2) - k{x 2 ,xi) 

to X determined by a feature map <i> : A —>■ iJi. In particular, Fortet’s condition 
then becomes: for all e > 0 there exists a countable partition Bj^j G N of A such 
that 

d$(a:i,X2) < \/e, Xi,X2 G Bj,j gN . (2.1) 

We begin with a preparatory lemma asserting that the separability of the image of 
the feature map implies the separability of the corresponding RKHS or RKBS. This 
lemma is used in both the proof of our generalization of Fortet’s result. Theorem 
2.2, which is valid when A is not a topological space, and our main result Theorem 
2.4, valid when A is a separable absolute Borel space or an analytic subset of a 
Polish space. 

Lemma 2.1. Consider a (RKBS) RKHS K, of functions on a set X with feature 
(Banach) Hilbert space W and (primary) feature map $ : A —>■ W. If ‘i>(A) C W 
is a separable subspace, then K is separable. 

We can now present our generalization of Fortet’s result to RKBSs expressed in 
terms of the pseudometric space (A, (i$)^. 

^We would like to thank one of the referees for pointing out the possible connection between Fortet’s 
condition (2.1) and a Lindelof type condition on the pseudometric space (A, d<s,) 
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Theorem 2.2. A RKBS K. of functions on a set X is separable if and only if 
there exists a feature Banach space W and feature map $ : X —>■ W such that the 
topological space {X^dg,) determined by the pullback pseudometric 

dg,{xi,X2) ■■= ||5>(a;i) - $(a:2)||w, Xi,X2 € X 


is separable. 

Now let us consider the case when X is a topological space. Since separability is 
preserved under continuous maps, see e.g. [39, Thm. 16.4], Lemma 2.1 implies the 
RKBS version of Steinwart and Christmann [33, Lem. 4.33] when combined with 
[33, Lem. 4.29]: A RKBS of functions on a separable space X is separable if it has 
a continuous feature map. Steinwart and Christmann [33, Lem. 4.33] assert that if 
X is separable and the kernel k corresponding to the RKHS H is continuous, then 
H is separable. More generally, Steinwart and Scovel [36, Cor. 3.6] show that if 
there exists a finite and strictly positive Borel measure on X, then every bounded 
and separately continuous kernel k has a separable RKHS. However, to obtain our 
main result, our primary tool to derive separability comes from theorems of Stone 
[37, Thm. 16, pg. 32], when X is separable absolute Borel, and Srivastava’s [31, 
Thm. 4.3.8] version of Simpson [29] when X is an analytic subset of a Polish space. 
It is interesting to note that Srivastava’s proof is different from Simpson’s in that it 
does not use Stone’s Theorem [37, Thm. 16, pg. 32]. 

Lemma 2.3. Let X be separable absolute Borel or an analytic subset of a Polish 
space and let Y be a metric space, and suppose that f : X ^ Y is Borel measurable. 
Then f{X)cY is separable. 

Steinwart and Christmann [33, Lem. 4.25] shows that separate measurability of 
the kernel combined with separability of the corresponding RKHS implies that the 
canonical feature map is measurable. Our main result is a kind of converse when X 
is separable absolute Borel or an analytic subset of a Polish space. 

Theorem 2.4. Let X be separable absolute Borel or an analytic subset of a Polish 
space and let K. be a RKHS with measurable feature map, or a RKBS with mea¬ 
surable primary feature map, of real-valued functions on X. Then K is separable. 


3 Proofs 

3.1 Proof of Lemma 2.1 

For RKHSs this assertion is contained in the proof of Steinwart and Christmann [33, 
Lem. 4.33]. Roughly, the argument is that rational linear combinations are dense in 
the linear span of 4’(X) and the linear span is dense in the closed linear span in the 
metric defined in the proof of [33, Thm. 4.21]. For the RKBS case, since $ : X —W 
is a primary feature map it satisfies span(4>(X)) = W. Moreover, since $(X) C W is 
separable, the same argument as used in the RKHS case shows that the closed linear 
span span($(X)) = W is separable, so we conclude that W is separable. Since W 
is reflexive it follows from [26, Cor. 1.12.12] that W* is separable. Moreover, Zhang, 
Xu and Zhang [40, Thm. 3] implies that the dual Banach space is 


1C* :={[$(•),«*] eW*} 
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with norm 

||[$(.),«1lk. := IKIIw, 

so that the mapping from W* to /C* defined by u* ^ [<&(•), u*] is an isometry. 
Consequently, the separability of W* implies the separability of /C*. Since /C and 
therefore /C* are reflexive it follows from [26, Cor. 1.12.12] that /C is separable. 

3.2 Proof of Theorem 2.2 

Let us first demonstrate the equivalence between the separability of the RKBS and 
Fortet’s condition (2.1). Then we will demonstrate the equivalence between Fortet’s 
condition and the separability of the pseudometric space {X, d$). Let us begin with 
”if’. To that end, let us show that condition (2.1) implies that $(X) is separable. 
Indeed, fix e > 0 and for each fc G N let i?^, j G N denote the corresponding 
partition and let Xj G Bj denote a selection. Then the set k G J\f,j G Af is 

countable, and it is easy to show it is dense in $(X). That is, $(X) is separable, 
and the separability of /C follows from Lemma 2.1. Now for the ’’only if’, suppose 
that /C is separable. Then the canonical feature space W := K. is separable, and 
since /C is metric, by e.g. [39, Thm. 16.8] it is second countable. Therefore, since 
second countability is inherited by subspaces, see e.g. [39, Thm. 16.2], it follows for 
the corresponding canonical feature map $ : X —>• /C, that $(X) C /C is second 
countable, and therefore, by e.g. [39, Thm. 16.9], it is separable. Therefore there 
exists a countable dense set ^{xj) G <^(X),j G N. Therefore, if for each e > 0 
and for each j G N we define Bj = {x G X : ||$(a;j) — $(a;)|k < §}, it follows 
that UjgpjBj = X and ||$(xi) — $(a:2)|k < £ foi' xi,X2 G Bj. Therefore, we 
have established the equivalence between the separability of the RKBS and Fortet’s 
condition (2.1). 

Now let us demonstrate the equivalence between Fortet’s condition and the sepa¬ 
rability of the pseudometric space (X, d$). To that end, suppose that the pseudomet¬ 
ric space (X, (i$) is separable. Then Willard [39, Thm. 16.11] asserts that in a pseu¬ 
dometric space, the conditions of being Lindelof, second countable, and separable are 
equivalent. Therefore (X, (i$) is Lindelof in that every open cover has a countable 
subcover. For x G X, let B^{x,e) := {x' G X : d^{x,x') < e} denote the open ball 
about x, and for each e > 0 consider the open cover {R$(a:, ^),x G X}. Then since 
(X, (i$) is Lindelof it follows there exists a countable subcover {i?$(a;, |),a; G Xq} 
where Xq is a countable. This cover satisfies Fortet’s condition (2.1) for the value 
e and since e was arbitrary, it follows that the map $ : X —?> W satisfies Fortet’s 
condition (2.1). In the other direction, suppose that the map $ : X —>■ W sat¬ 
isfies Fortet’s condition (2.1). Fix e > 0 and for each G N let B^^j G N 
denote the corresponding partition and let Xj G Bj denote a selection. Then the set 
$(a:^), k G Af,j G Af is countable, and it is easy to show it is dense in $(X). That 
is, for X G X, the countable set {^{xj),k G Af,j G Af} comes arbitrarily close to 
$(a:). It follows that the countable set {Xj,k G Af,j G Af} comes arbitrarily close 
to X in the pseudometric d<i>. Consequently (X, (i$) is separable. 

3.3 Proof of Lemma 2.3 

The case when X is an analytic subset of a Polish space follows directly from Srivas- 
tava [31, Thm. 4.3.8]. When X is separable absolute Borel, it follows from Stone’s 
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Theorem [37, Thm. 16, pg. 32] that when T is a metric space and $ : X —>■ T 
is a measurable bijection, that the image Y is separable. However, when $ is not 
surjective, since $(X) C H is a metric space, the assertion that the metric sub¬ 
space ‘h(X) C Y is separable follows assuming that $ is a measurable injection. 
Moreover, injectivity is also unnecessary. To see this, extend to the injective map 
^ : X ^ X X Y defined by 4(x) := (cc, <i>(a:)). Then it follows from Hansell’s [21, 
Thm. 1] generalization of Kuratowski [24, Thm. 1, Sec. 31, VI] to the nonseparable 
case, that $ is measurable. To see how it is obtained, since X is assumed to be sepa¬ 
rable and metrizable, it is second countable, see e.g. [39, Thm. 16.11], so that it has 
a countable base {Gn, n G N} of open sets generating its topology. Let W d X xY 
be open and define 


14 = u{V : V open, G„ x V C IT} . 


Then 


and therefore 


IT = U G„ X T„ 

n6N 

$- 1 ( 14 ) = u G„n$-1(T„). 

n^N 


Since Gn and 14 are open and therefore measurable and $ is measurable it follows 
that <1>-1(1T) is measurable. Consequently, since the open sets generate the Borel 
tr-algebra, it follows that $ is Borel measurable. Moreover, since $ is injective the 
above discussion shows that $(V) C V x T is separable. Since 4’(V) = Py$(X) 
where Py is the projection onto the second component and Py is continuous, and 
separability is preserved under continuous maps, see e.g. [39, Thm. 16.4], it follows 
that $(V) C T is separable. 


3.4 Proof of Theorem 2.4 


Since the feature space is metric. Lemma 2.3 implies that the image 4’(V) is sepa¬ 
rable for any measurable feature map $. The assertion then follows from Lemma 


2 . 1 . 
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